
Tools 


Guide to the Competency-based 
Learning Survey for Students 


Sarah Ryan 
Joshua D. Cox 

Education Development Center, Inc. 

In collaboration with the 
Northeast College and Career Readiness Research Alliance 


:*REL 

NORTHEAST 
S> ISLANDS 


ries 


NATIONAL CENTER for 
EDUCATION EVALUATION 
and REGIONAL ASSISTANCE 


Regional Educational Laboratory 
At Education Development Center, Inc. 


August 2016 




U.S. Department of Education 

John B. King, Jr., Secretary 

Institute of Education Sciences 

Ruth Neild, Deputy Director for Policy and Research 
Delegated Duties of the Director 

National Center for Education Evaluation and Regional Assistance 

Joy Lesnick, Acting Commissioner 
Amy Johnson, Action Editor 
Elizabeth Eisner, Project Officer 

REL 2016-165 

The National Center for Education Evaluation and Regional Assistance (NCEE) conducts 
unbiased large-scale evaluations of education programs and practices supported by federal 
funds; provides research-based technical assistance to educators and policymakers; and 
supports the synthesis and the widespread dissemination of the results of research and 
evaluation throughout the United States. 

August 2016 

This report was prepared for the Institute of Education Sciences (IES) under Contract 
ED-IES'12'C-0009 by Regional Educational Laboratory Northeast & Islands administered 
by Education Development Center, Inc. The content of the publication does not neces- 
sarily reflect the views or policies of IES or the U.S. Department of Education, nor does 
mention of trade names, commercial products, or organizations imply endorsement by the 
U.S. Government. 

This REL report is in the public domain. While permission to reprint this publication is 
not necessary, it should be cited as: 

Ryan, S., & Cox, J. (2016). Guide to the Competency'based Learning Survey for Students 
(REL 2016-165). Washington, DC: U.S. Department of Education, Institute of Education 
Sciences, National Center for Education Evaluation and Regional Assistance, Regional 
Educational Laboratory Northeast & Islands. Retrieved from http://ies.ed.gov/ncee/edlabs. 

This report is available on the Regional Educational Laboratory website at http://ies.ed.gov/ 
ncee/edlabs. 



Summary 


Many states are moving away from approaches that base student advancement on credits 
and “seat time” toward competency-based learning approaches that provide schools with 
the flexibility to link a student’s advancement to mastery of content. Regional Educational 
Laboratory Northeast & Islands, in partnership with the Northeast College and Career 
Readiness Research Alliance, has developed a new survey — the Competency-based Learn- 
ing Survey for Students — to help high schools and school districts collect data on student 
experiences with competency-based learning . 1 Policymakers and practitioners need this 
information to improve implementation of and communication about competency-based 
learning with students. 

The survey is designed to be administered to students attending high schools in which 
competency-based learning is being implemented. It collects information on students’ 
beliefs about, exposure to, and understanding of key elements of competency-based learn- 
ing. These elements include instructional practices that allow students to progress through 
demonstration of mastery, receive personalized instruction and learning opportunities, 
demonstrate mastery through flexible assessment, and develop specific skills and disposi- 
tions that may be especially critical under competency-based learning. 

The guide describes the survey in greater detail, including why it was developed, how to 
administer and adapt it, and how to analyze the results. The guide also includes the version 
of the survey that was developed for use with high school students in New Hampshire. 
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What is the Competency-based Learning Survey for Students? 


In partnership with the Northeast College and Career Readiness Research Alliance, 
Regional Educational Laboratory (REL) Northeast & Islands developed the Competency' 
based Learning Survey for Students to help high schools and school districts collect data 
on student experiences with competency-based learning. The survey is designed to be 
administered to students attending high schools in which competency-based learning is 
being implemented. It collects information on students’ beliefs about, exposure to, and 
understanding of key elements of competency-based learning. These elements include 
instructional practices that allow students to progress through demonstration of mastery, 
receive personalized instruction and learning opportunities, demonstrate mastery through 
flexible assessment, and develop specific skills and dispositions that may be especially criti- 
cal under competency-based learning. 

Why was the survey developed? 


Many states are moving away from approaches that base student advancement on credits 
and “seat time” toward competency-based learning approaches that provide schools with 
the flexibility to link advancement to a student’s mastery of content (Scheopner Torres, 
Brett, & Cox, 2015). Many terms are used to describe approaches to teaching and learn- 
ing in which students advance after demonstrating mastery, including competency-based 
learning, competency-based education, proficiency-based learning, mastery-based learning, 
and standards-based learning. This guide uses the term competency-based learning. 

The movement toward competency-based approaches addresses the recommendation from 
the National Center on Time & Learning (2011) that learning programs should account 
for the different amounts of time student populations need to achieve proficiency, and 
should consider “advancement based on mastery, rather than ‘seat time’” (p. 37). This shift 
also reflects the position that students are more engaged in their learning when they are 
granted greater flexibility in how they earn credit (U.S. Department of Education, 2011). 

The seven states in the REL Northeast & Islands Region (Connecticut, Maine, Massachu- 
setts, New Hampshire, New York, Rhode Island, and Vermont) are actively exploring ini- 
tiatives designed to increase high school completion rates and ensure student readiness for 
college and careers. Since 2003 Maine, New Hampshire, Rhode Island, and Vermont have 
made policy changes to encourage implementation of competency-based learning across 
their states by requiring all school districts to make high school graduation contingent on 
a student’s mastery of all required competencies. School districts in Connecticut, Massa- 
chusetts, and New York can choose to implement one or more elements of competency- 
based learning but are not required to do so. 

Previous research done in collaboration with the Northeast College and Career Readi- 
ness Research Alliance suggests that the nature of implementation varies across states, as 
well as among school districts within a state (Scheopner Torres et al., 2015). Within and 
across states, school districts vary in the extent to which they base student progression 
on demonstration of mastery rather than fulfillment of seat time requirements. School 
districts also vary in how much they have incorporated personalization practices, flexible 
assessment, and development of skills and dispositions that promote student success under 
competency-based learning. Some variability is driven by implementation challenges, 
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which include understanding and addressing student misconceptions about competency- 
based learning, helping students take greater ownership over their own learning under 
competency-based learning, and carrying out necessary yet substantial shifts in instruc- 
tion, assessment, and grading in the classroom (Scheopner Torres et al, 2015). 

Currently, few high schools or school districts systematically collect information on how 
much competency-based learning approaches are being implemented. Alliance members 
have indicated that more research is needed to help states implement competency-based 
learning successfully. Specifically, regional stakeholders have reported that information on 
students’ beliefs about, exposure to, and understanding of key elements of competency- 
based learning could help policymakers and practitioners in the REL Northeast & Islands 
Region and elsewhere improve implementation and communication with students. As a 
result, alliance members requested support in developing a survey to measure students’ 
beliefs about, exposure to, and understanding of key elements of competency-based learn- 
ing (box 1). Alliance members specifically desired a survey that could be used in high 
schools and school districts that have adopted some form of competency-based learning, 
including early and more advanced stages of implementation. 

The need expressed by stakeholders in the REL Northeast & Islands Region aligns with 
national research underscoring the importance of examining students’ perceptions of and 
experiences with competency-based learning (Lewis et al., 2014; Steele et al., 2014). Recent 
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Box 1. Key elements of competency-based learning 

Rather than a single approach, competency-based learning comprises multiple elements. 
A growing research consensus suggests that key elements of competency-based learning 
include progression through demonstration of mastery, personalization, flexible assessment, 
and development of specific skills and dispositions (Patrick & Sturgis, 2011, 2013; Scheopner 
Torres et al., 2015; Steele et al., 2014). These elements include: 

• Progression through demonstration of mastery requires a student to demonstrate that he or 
she has learned what was expected before moving on to the next level. 

• Personalization refers to the provision of individualized support, flexible pacing, and oppor- 
tunities for student choice in how to demonstrate mastery. 

• Flexible assessment includes exposure to multiple modes of assessment, which allows 
students to demonstrate mastery in a variety of ways rather than through only one assess- 
ment (for example, a written test). 

• Development of specific skills and dispositions such as perseverance and a capacity to 
self-direct one’s learning may be especially important in a competency-based learning 
system where student agency and choice are emphasized (Lewis et al., 2014). 

States and school districts in the Regional Educational Laboratory Northeast & Islands 
Region implementing competency-based learning also identify taking responsibility for one’s 
work, collaboration, and respect for others as important student skills and dispositions. 

While there is general agreement about key elements of competency-based learning, its 
implementation varies substantially across states and school districts (Scheopner Torres 
et al., 2015; Steele et al., 2014). Competency-based learning policies and practices can range 
from reforms that simply allow flexibility in awarding credit to reforms that completely trans- 
form the education system “from a time-based system to a learning-based system” (Sturgis & 
Patrick, 2010, p. 4). 
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research suggests that differential access to learning experiences under competency-based 
learning could produce differential levels of learning (Lewis et al., 2014). With information 
on students’ exposure to and understanding of competency-based learning, school and dis- 
trict policymakers and practitioners could examine potential differences in students’ expe- 
riences and how such differences may be related to student outcomes (Lewis et al., 2014; 
Steele et al, 2014). 

Based on the need expressed by stakeholders and the recent research supporting this need, 
the study team formed and worked closely with an advisory committee to develop the 
Competency-based Learning Survey for Students. The committee included six members: 
two teacherdeaders 2 from a Connecticut school district, a researcher from a Maine univer- 
sity, a high school teacher (also a former researcher) from Maine, a district administrator 
from New Hampshire, and a high school principal from Vermont. All advisory committee 
members worked in or with schools that were at various stages in the implementation of 
competency-based learning. 


About the survey 


Potential survey users include administrators, teachers, and other staff in high schools that 
have already implemented or are beginning to implement competency-based learning and 
desire information on how students understand and experience this reform. The survey is 
intended to provide these high schools and school districts with a resource for measuring 
students’ beliefs about, exposure to, and understanding of key elements of competency-based 
learning. For example, survey results will provide information on what students believe about 
specific instructional policies and practices associated with competency-based learning, 
whether students understand the school or district rationale for implementing specific policies 
and practices, and the extent to which students are exposed to specific policies and practices. 

The survey includes 30 items (questions or statements) and their respective subitems. It 
was designed using an iterative process (see appendix A) and allows users to collect infor- 
mation on several constructs (an abstract idea, underlying theme, or subject matter that 
is represented by several individual survey items), represented by groups of survey items 
organized into an introductory section and six survey modules (table 1). 

Because of the wide variability in implementation of competency-based learning, survey 
users may select which modules to administer based on the elements of competency-based 
learning that have been implemented in their local context. Module A collects demo- 
graphic information to help users understand the student population taking the survey. 
Module B addresses students’ beliefs and understanding about competency-based learning. 
With information from module B, school or district administrators and staff can gauge the 
extent to which student understanding is consistent with the school’s or district’s rationale 
for implementing different elements of competency-based learning. These data can also 
help identify specific elements of competency-based learning about which students may 
need additional information. For example, the results may suggest a need for clearer com- 
munication with students about the rationale behind specific competency-based learning 
policies and practices (for example, why homework is not always graded). 

Each of the remaining four modules (modules C-F) collects data on students’ exposure to 
a key element of competency-based learning: progression through demonstration of mastery, 
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Table 1. Survey constructs, modules, and items of the Competency-based Learning Survey for 

Students 

Construct 

Description 

Module 

Items 

Approximate 
time to complete 

Student demographics 

Demographics of the responding student 

A 

A1-A7 

1 minute 

Student beliefs about 

competency-based learning 

Beliefs about specific aspects of schooling that are likely to be 
influenced by the implementation of competency-based learning 

B 

B1 

2 minutes 

Student understanding of 
competency-based learning 

Understanding of the rationale behind specific school policies 
or practices that reflect key elements of competency-based 
learning 

B 

B2-B14 

5 minutes 

Progression through 
demonstration of mastery 

Exposure to explanation of the competencies and how to 
demonstrate progress toward mastery of competencies 

C 

C1-C2 

2 minutes 

Personalization 

Exposure to multiple options for earning course credit as well 
as personalized instruction and learning opportunities 

D 

D1-D3 

3 minutes 

Flexible assessment 

Exposure to flexible assessment 

E 

E1-E2 

2 minutes 

Development of skills and 
dispositions 

Exposure to opportunities to develop skills and dispositions 
including respect for others, perseverance, self-directedness, 

and collaboration 

F 

F1-F2 

3 minutes 

Source: Authors' compilation. 


personalization, flexible assessment, and development of specific skills and dispositions. With 
data from these modules, school or district administrators and staff can identify how well stu- 
dents’ reported exposure to elements of competency-based learning aligns with how the school 
or district intends the elements to be implemented. For example, if many students report that 
instruction includes mostly whole-class lecture by the teacher or that students mostly work 
on the same assignment at the same time, this response pattern might suggest that students 
experience little exposure to personalization. Data from these modules could also be used to 
examine relationships among students’ beliefs about elements of the approach, students’ expo- 
sure to specific elements of the approach, and students’ understanding of the rationale behind 
why the school uses those elements. 

Further, if combined with existing student data, the survey data could be used by school 
and district administrators to examine associations among students’ beliefs about compe- 
tency-based learning, students’ exposure to specific elements of competency-based learn- 
ing, students’ understanding of competency-based learning, and students’ outcomes. 

Finally, the survey was developed to be used in high schools at various stages of implementa- 
tion of competency-based learning. In schools at earlier stages of implementation, students 
are likely to experience varying degrees of competency-based learning policies and practic- 
es across their courses. Capturing this variability through the survey will provide schools 
with a snapshot of schoolwide implementation. Further, administering the survey annually 
will allow schools to observe, at the school level, how students’ beliefs about, exposure to, 
and understanding of competency-based learning policies and practices change over time. 
This may be especially useful in schools at early stages of implementation. 

How to administer the survey 
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The Competency-based Learning Survey for Students can be adapted to fit the local 
context. While it can be distributed to students as a paper-and-pencil survey, it is intended 
to be administered online, which makes it easier for schools to administer the survey and 
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calculate descriptive results. Online administration also reduces the likelihood of errors 
during administration and analysis. It should take students about 15-20 minutes to com- 
plete the survey when all six modules are administered. 

How to adapt the survey 


Survey users may adapt any part of the survey for their own use. For example, the survey 
presented in this document was tailored for collecting data from high school students in 
New Hampshire and thus uses language specific to that state. 

In adapting the survey, users should first revise the introduction so that it describes the 
purpose of the survey, why it is important for students to complete, and how the data will 
be used. Second, users should review the survey items to ensure that the terminology will be 
familiar to students in the local context. For example, New Hampshire high schools use the 
term competency-based learning, and in many schools in the state, staff and students regu- 
larly talk about the competencies that students must master as well as the competency-based 
grading system. In contrast, the term proficiency-based learning is used in Maine, and many 
Maine high school students have the option to earn a proficiency-based diploma. While staff 
and students in some schools may talk about demonstrating progress on standards (rather 
than about mastering competencies, as in New Hampshire), staff do not necessarily use the 
language of standards with students in all schools. Instead, school staff may use more indirect 
language to describe expectations for students — for example, by referring to the importance 
of learning the material covered in the course or of completing lessons and units. 

Variability in the terminology related to competency-based learning — as well as variability 
in whether and how specific terms are used with students — means that survey users must 
carefully consider the terminology used in their locales and adapt the survey introduction 
and relevant survey items accordingly. To illustrate how the same survey items might need 
to be revised for use in different contexts, examples of select survey items adapted for use 
in New Hampshire and in Maine are shown in table 2. 3 

The survey was designed such that all respondents would review the survey introduction 
and complete module A. However, users should determine which of the other modules are 
appropriate based on the elements of competency-based learning that have been imple- 
mented in their local context. 

Since module B collects information on students’ beliefs about competency-based learning 
(see item Bl) and students’ understanding of the rationale behind elements of competency- 
based learning that have been implemented at their school, module B is likely to be appro- 
priate for students in most contexts. For each of the items related to student understanding 
(items B2-B14) students are first asked whether a specific policy or practice is used at their 
school. Students who respond affirmatively (for example, “yes,” “often,” or “sometimes”) are 
routed to an item that asks them to select the response that they believe best describes why 
the policy or practice is used at their school. Students who respond “no,” “never,” or “don’t 
know” are routed to the next relevant survey item using survey skip logic. 

Modules C-F each focus on a specific element of competency-based learning, and users 
should select from these modules based on which elements are being implemented in their 
local context. For example, if implementation of competency-based learning in the local 
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Table 2. Comparison of select survey items adapted for use in New Hampshire and Maine 


1 Module 

Item 

New Hampshire 

Maine 1 

B 

B8 

Do any of the courses at your school use a competency- 
based grading system? 

Do any of the courses at your school use a proficiency- 
based grading system? 

B 

B12 

At this school, when teachers talk about competencies, 
they are referring to: 

The courses a student must take to graduate 

The important skills and knowledge a student must 
learn to graduate 

The required credits a student must complete to 
graduate 

I’m not sure what teachers mean when they talk 
about competencies 

Teachers do not talk about competencies 

At this school, when teachers talk about standards, they 
are referring to: 

The courses a student must take during high school 

The important skills and knowledge a student must 
learn during high school 

The required credits a student must complete during 
high school 

I'm not sure what teachers mean when they talk 

about standards 

Teachers do not talk about standards 

C 

Cl 

1 must show my teachers that 1 have mastered each 
competency before 1 can move on to the next one 

1 must show my teachers that 1 am ready for new 
material before 1 can begin a new lesson or unit in my 

courses 

C 

C2 

1 understand how the competencies in my courses will 
help me in the future. 

1 understand how the material 1 am studying in my 
courses will help me in the future 

Source: Authors' construction based on versions of the Competency-based Learning Survey for Students administered in New Hamp- 
shire and Maine. 


context involves progression through demonstration of mastery and flexible assessment 
but does not involve a focus on personalization or on specific skills and dispositions, users 
would not administer module D or module F. 

Finally, where survey items refer to students’ courses, students are asked to reflect on all 
of their courses at the school. If a school intends to implement competency-based learn- 
ing only in a specific course or in specific courses, any survey items referring to a student’s 
courses will need to be revised accordingly — for example, by using language such as “in this 
course.” Similarly, survey items that refer to a particular timeframe, such as a semester, may 
need to be revised to reflect the timeframe of interest at the school. Further, administering 
the survey at the beginning of a particular period, such as a school year, might pose chal- 
lenges because students will have limited experience in their current courses to report on. 

How to analyze the survey results 


Survey users can calculate descriptive statistics, including frequencies and cross-tabula- 
tions of item responses. Each survey item requires respondents to select from one of several 
options. Some items require students to select from response options that have no natural 
ordering: A2-A4, A7, B3, B5, B7-B9, and B11-B14. These items are best summarized by 
calculating the number and percentage of respondents that select each option. Examples 
of how results for this type of item might be presented as a figure or a table are provided in 
figure B1 and table B1 in appendix B. 
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Other items require respondents to select from options that share a natural ordering: Al, 
A5, A6, Bl, B2, B4, B6, BIO, Cl, C2, D1-D3, El, E2, FI, and F2. These items can be sum- 
marized by calculating the number and percentage of respondents who select each option 
or by using measures of central tendency (for example, means and medians) and variability 
(variances and standard deviations). This class of items includes Likert-type items, which 
typically assume a negative-to-positive dimension underlying the discrete (often, five to 
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seven) response options — for example, a dimension ranging from strong agreement to 
strong disagreement. Item B1 is an example of a Likert-type item. 

An item that uses a categorical frequency scale also requires respondents to select from 
options that share a natural ordering. Items that use a categorical frequency scale assume 
an infrequent-to-very-frequent dimension underlying the discrete response categories — for 
example, a dimension ranging from never to always. Examples of how results for an item 
that uses a categorical frequency scale might be presented as a figure or table are provided 
in figure B2 and table B2 in appendix B. 


Researchers often analyze Likert-type items and items that use a categorical frequency 
scale by treating the response options as if they were based on an interval scale (a scale 
in which there is an equal distance between response options). For both Likert-type items 
and items that use a categorical frequency scale, the study team advises that if means and 
standard deviations (interval scale statistics) are reported for individual items, the percent- 
age or frequency of students who selected each option should also be reported to support 
informed interpretation of the results at the item level. In addition, respondents who select 
“not sure” on these item types should be analyzed separately (that is, not included in the 
calculation of means) because this response would not fall along the natural ordering of 
the responses. 

The survey was tested with a small sample of high school students (see appendix A), but 
users are encouraged to conduct additional testing and to report those results. For example, 
before administering the survey statewide a state could administer the survey in a subset 
of school districts to determine whether the survey is working as intended. Similarly, a 
district could first test the survey with a subset of high schools (or with a subset of students 
in school districts with only one high school). The survey should not be used to evaluate 
student or teacher performance or to inform decisions about funding for competency-based 
learning. Data should be evaluated in aggregate to inform school- or district-level deci- 
sions on topics such as communication with students about competency-based learning or 
implementation of different elements of competency-based learning. 
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The Competency-based Learning Survey for Students 


The remainder of this guide presents the Competency-based Learning Survey for Students 
used in New Hampshire. 

Introduction 

This is a survey about your experiences at this school and in your courses at this school. 
All of your answers will be confidential. 

This survey is NOT A TEST and there are no right or wrong answers. We really hope 
you will try to answer every question, but you may skip any question you do not wish to 
answer. Your responses will be collected through a secure survey system and all data will be 
encrypted and stored on secure servers. All information from this survey will be reported 
about your school or district as a whole. No one will be able to connect individual respons- 
es to you or to any other student. 

It should take about 15-20 minutes to complete the survey. By clicking on “NEXT” you 
agree to participate in this study. 

Module A: Student demographics 

Al. What is your current grade level? 

□ Grade 9 

□ Grade 10 

□ Grade 11 

□ Grade 12 

A2. Are you male or female? 

□ Male 

□ Female 

□ Prefer not to say 

A3. Is this your first year in this high school? 

□ Yes 

□ No 

A4. Did you attend school in this school district last year? 

□ Yes 

□ No 

A5. What is your current high school grade point average (GPA)? 

□ 0 . 00 - 1.00 

□ 1 . 01 - 2.00 

□ 2.01-3.00 

□ 3.01-4.00 

□ 4.01-5.00 

□ Not sure 
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A6. Currently, how many of your courses are you taking as online courses? An online 
course is one in which you complete all instruction and assignments over the 
Internet. 

□ None 

□ 1-2 

□ 3 or more 

A7. How much education do you expect to complete? 

□ Less than high school 

□ High school diploma 

□ GED 

□ Associate’s degree 

□ Some college but less than a bachelor’s degree 

□ Bachelor’s degree 

□ Master’s degree 

□ Doctoral or advanced professional degree 

Module B: Student beliefs and understanding 

This set of questions asks about practices and requirements at your school. Your responses 
to these questions will help school leaders and teachers understand students’ experiences 
at your school. Like all questions in this survey, there are no right or wrong answers to 
these questions. 

Bl. For the following statements, indicate how much you agree or disagree. 



Strongly 


Neither agree 


Strongly 

Subitem 

disagree 

Disagree 

nor disagree 

Agree 

agree 

The graduation requirements at my school will 
prepare me for what 1 want to do after high school 

□ 

□ 

□ 

□ 

□ 

Students should get more than one opportunity 
to show whether they have learned the important 

course material 

□ 

□ 

□ 

□ 

□ 

Students should get more than one opportunity to 
pass a test or exam 

□ 

□ 

□ 

□ 

□ 

Homework is important to complete even if it is not 
graded 

□ 

□ 

□ 

□ 

□ 

If two students in the same course complete 
different types of assignments, they should still 
have the opportunity to earn the same grade in the 

□ 

□ 

□ 

□ 

□ 

course 






My grades are a good reflection of what 1 have 

learned 

□ 

□ 

□ 

□ 

□ 

Most colleges will know what my course grades 
mean. In other words, most colleges will know how 
to evaluate the grades I’ve received in the courses 

I’ve taken at my school 

□ 

□ 

□ 

□ 

□ 
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For some of the next several items, several options may seem SOMEWHAT TRUE. 
However, you should choose the response that is the MOST TRUE at your school. 

B2. In courses at my school, students must show their learning on each competency in 
more than one way. For example, students must show that they have mastered the 
competency on more than one assignment, assessment, or exam. [Skip logic will be 
used.] 

□ Never (Skip to item B4) 

□ Sometimes 

□ Often 

□ Don’t know (Skip to item B4) 

B3. Why do you think students must show their learning on each competency in more 
than one way? Choose the response that seems MOST TRUE about this school. 

□ A student’s grade in the course should be based on multiple scores 

□ Teachers need multiple opportunities to see whether students have learned the 
material 

□ Students stay on task better if they have enough work to do 

□ Completing multiple assignments and tests helps prepare students for the state 
standardized test 

B4. In courses at my school, students are able to choose how they want to show what 
they have learned from several different options. For example, options such as taking 
a test, writing a paper, completing a project, etc. [Skip logic will be used.] 

□ Never (Skip to item B6) 

□ Sometimes 

□ Often 

□ Don’t know (Skip to item B6) 

B5. Why do you think students in courses at this school are able to choose how they 
want to show their learning? Choose the response that seems MOST TRUE about 
this school. 

□ Most students know how they learn best 

□ Some students aren’t good at taking tests/exams 

□ Each student should be able to get an A in the course 

□ It is good for students to have variety in their work 

B6. Students at my school are able to progress at their own individual pace in courses. 
[Skip logic will be used.] 

□ Never (Skip to item B8) 

□ Sometimes 

□ Often 

□ Don’t know (Skip to item B8) 

B7. Why do you think students in courses at this school are able to progress at their own 
individual pace? Choose the response that seems MOST TRUE about this school. 

□ Some students fall behind if they have been absent a lot 

□ Some students are less interested in certain topics 

□ Some students may need different amounts of time to learn the material 

□ Some students don’t complete all of their work on time 
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B8. Do any of the courses at your school use a competency-based grading system? [Skip 
logic will be used.] 

□ Yes 

□ No (Skip to item BIO) 

□ Don’t know (Skip to item BIO) 

B9. Why do you think all or some of the courses at your school use a competency-based 
grading system? Choose the response that is MOST TRUE at your school. 

□ Teachers do a better job of grading student work under a competency-based 
grading system 

□ Students take fewer tests under a competency-based grading system 

□ A competency-based grading system makes it easier for all students to graduate 
from high school 

□ A competency-based grading system provides better information about what a 
student has learned 

BIO. Students at my school are assigned homework. [Skip logic will be used.] 

□ Never (Skip to item B12) 

□ Sometimes 

□ Often 

□ Don’t know (Skip to item B12) 

Bll. Why do you think students at this school are assigned homework? Choose the 
response that seems MOST TRUE at this school. 

□ Homework provides students with opportunities to practice a skill before being 
assessed on that skill 

□ Students in high school should have homework almost every night to learn the 
material 

□ Teachers don’t have time to teach all of the important material during class 

□ Completing several hours of homework most nights helps students get ready for 
college 

B12. At this school, when teachers talk about competencies they are referring to: 

□ The courses a student must take to graduate 

□ The important skills and knowledge a student must learn to graduate 

□ The required credits a student must complete to graduate 

□ I’m not sure what teachers mean when they talk about competencies 

□ Teachers do not talk about competencies 

B13. Do you have to master specific competencies in any of your courses in order to grad- 
uate from your high school? [Skip logic will be used.] 

□ Yes 

□ No (Skip to next module) 

□ Don’t know (Skip to next module) 

B14. Has a principal or teacher explained to you why it is important to master specific 
competencies to graduate from your high school? 

□ Yes 

□ No 

□ Don’t know 
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Module C: Progression through demonstration of mastery 


In this section of the survey, you will answer some questions related to courses at this 
school, including how students make progress in their learning. 

Cl. For these items, indicate about how often each of the following happens in your 
courses at this school. 


Subitem 

Never 

Seldom 

Sometimes 

Often 

Always 

Not sure 

1 know what 1 need to do to show my teachers 
that 1 am making progress on each competency 

□ 

□ 

□ 

□ 

□ 

□ 

1 must show my teachers that 1 have mastered 
each competency before 1 can move on to the 

next one 

□ 

□ 

□ 

□ 

□ 

□ 

1 am able to move on to the next competency 
when 1 am ready, even if other students in the 
course are not ready 

□ 

□ 

□ 

□ 

□ 

□ 

Students in my courses work on the same 
competency at the same time 

□ 

□ 

□ 

□ 

□ 

□ 

C2. Indicate about how often each of the following happi 

ens in your 

courses at 

this 


school. 







Subitem 

Never 

Seldom 

Sometimes 

Often 

Always 

Not sure 

1 understand how the competencies in my 
courses will help me in the future 

□ 

□ 

□ 

□ 

□ 

□ 

My teachers share examples of excellent work 
on each competency 

□ 

□ 

□ 

□ 

□ 

□ 

My teachers let me know how my work will be 
assessed or graded for each competency 

□ 

□ 

□ 

□ 

□ 

□ 

My teachers give me a rubric so that 1 know how 

1 am progressing on each competency 

□ 

□ 

□ 

□ 

□ 

□ 
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Module D: Personalization 


Now you will answer some questions about different options students have for earning 
credit in courses at this school. You will also answer questions about different ways that 
teachers work with students. Again, there are no right or wrong answers to these questions. 

Dl. Think about the courses you are taking right now. For about how many of your 
courses is each of the following options true? 


1 Subitem 

None 

Some 

All or most 

Not sure 

1 am able to complete some or all of the course 
requirements online 

□ 

□ 

□ 

□ 

If 1 complete a project that wasn't assigned at school 
but is related to a course 1 am taking, 1 can earn credit 
for the project in that course 

□ 

□ 

□ 

□ 

1 can earn credit for taking courses at another high 

school 

□ 

□ 

□ 

□ 

1 can earn credit for taking courses at a college (for 
example, “dual-credit courses") 

□ 

□ 

□ 

□ 

1 can earn credit for completing an internship or job- 
shadowing in the community 

□ 

□ 

□ 

□ 

D2. For these items, indicate about how often this happi 

ens in courses at this school. 


1 Subitem 

Never 

Seldom 

Sometimes 

Often 

Always 

Not sure 

Students in my courses all work on the same 
assignment at the same time 

□ 

□ 

□ 

□ 

□ 

□ 

My teachers spend most of class time giving a 
lecture or presentation to the whole class 

□ 

□ 

□ 

□ 

□ 

□ 

My teachers work with students in small groups 
or individually 

□ 

□ 

□ 

□ 

□ 

□ 

My teachers notice if 1 need extra help 

□ 

□ 

□ 

□ 

□ 

□ 

My teachers teach the material in several 
different ways in order to help students learn 

□ 

□ 

□ 

□ 

□ 

□ 


D3. For these questions, think about how often each event occurred during the current 


semester. 








1 2 

3 4 

5 or more 


Subitem 

Never 

times 

times 

times 

Not sure 

My teachers or a counselor/advisor discussed how 1 
am doing on each competency with me 

□ 

□ 

□ 

□ 

□ 

My teachers gave me written feedback on my work 

□ 

□ 

□ 

□ 

□ 

1 have had opportunities to choose how to show my 

teachers what 1 have learned 

□ 

□ 

□ 

□ 

□ 
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Module E: Flexible assessment 


Thanks for being honest in your answers to the survey questions. Remember, there are no 
right or wrong answers. In this section of the survey, you will answer some questions about 
how you are asked to show your learning in your courses. In other words, these questions 
ask about different kinds of assessments used at this school. 

El. For these questions, think about how often each event occurred during the current 
semester. 





1 2 


3 4 


5 or more 


Subitem 

Not at all 


times 

times 


times 

Not sure 

1 have created drawings or models to show what 1 

have learned 

□ 


□ 


□ 


□ 

□ 

1 have taken tests or quizzes to show what 1 have 

learned 

□ 


□ 


□ 


□ 

□ 

1 have given a performance to show what 1 have 
learned (for example, performing in a video or skit/ 
play, playing an instrument) 

□ 


□ 


□ 


□ 

□ 

1 have given a presentation to show what 1 have 

learned 

□ 


□ 


□ 


□ 

□ 

1 have completed a project at school to show what 1 

have learned 

□ 


□ 


□ 


□ 

□ 

1 have completed a project in the community to 

show what 1 have learned 

□ 


□ 


□ 


□ 

□ 

E2. For each item, indicate about how often this happi 

ens in 

the 

courses you are taking. 


Subitem 

Never Seldom 

Sometimes 

Often 

Always 

Not sure 

If 1 do poorly on an assessment on the first try, 1 
can try again later 

□ 

□ 


□ 


□ 

□ 

□ 

To show that 1 have mastered a course 

competency, 1 must demonstrate my learning in 
more than one way 

□ 

□ 


□ 


□ 

□ 

□ 


14 






Module F: Development of skills and dispositions 


This section includes a few more questions about teaching and learning at this school. 
Again, there are no right or wrong answers to these questions. 

FI. For each item, indicate about how often this happens in the courses you are taking. 


Subitem 

Never 

Seldom 

Sometimes 

Often 

Always 

Not sure 

Teachers encourage students to respect the 
feelings of others 

□ 

□ 

□ 

□ 

□ 

□ 

Teachers show or explain to students how to 
treat each other with respect 

□ 

□ 

□ 

□ 

□ 

□ 

Teachers explain to students how they can 
disagree with each other respectfully 

□ 

□ 

□ 

□ 

□ 

□ 

When 1 have trouble learning something new, 
my teachers give me advice and strategies that 
help me to keep trying 

□ 

□ 

□ 

□ 

□ 

□ 

My teachers notice when 1 take extra time and 
effort on something that is difficult for me 

□ 

□ 

□ 

□ 

□ 

□ 

If 1 get a low score on an assessment, my 
teachers help me figure out how 1 can still do 
well in the class 

□ 

□ 

□ 

□ 

□ 

□ 

F2. For each item, indicate about how often this happens in 

the courses you are taking. 



Subitem 

Never 

Seldom 

Sometimes 

Often 

Always 

Not sure 

Teachers show or explain strategies students 
can use to keep track of their progress on each 
of the competencies 

□ 

□ 

□ 

□ 

□ 

□ 

Teachers show or explain strategies students 
can use to complete all assignments and 

assessments on time 

□ 

□ 

□ 

□ 

□ 

□ 

Teachers encourage students to take 
responsibility for their work 

□ 

□ 

□ 

□ 

□ 

□ 

1 know which steps to take during high school 
in order to prepare for what 1 want to do after 1 
graduate 

□ 

□ 

□ 

□ 

□ 

□ 

If 1 need information that 1 don’t have in order to 

□ 

□ 

□ 

□ 

□ 

□ 

complete an assignment, 1 know where to get it 

If it is difficult for me to get an assignment done 
on my own, 1 know strategies 1 can use so that 

I’m sure to get the work finished on time 

□ 

□ 

□ 

□ 

□ 

□ 

Teachers show or explain strategies students 
can use to help each other learn 

□ 

□ 

□ 

□ 

□ 

□ 

Teachers show or explain strategies students 
can use to work together successfully in groups 

□ 

□ 

□ 

□ 

□ 

□ 

Teachers encourage students to help each 

other outside of class 

□ 

□ 

□ 

□ 

□ 

□ 
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Appendix A. Developing the survey 


The study team used seminal survey development resources, including Dillman, Smyth, 
and Christian (2008), Fowler (2008), Rea and Parker (2005), and Wright and Marsden 
(2010), to guide the development of the survey instrument. Survey development relied on 
close collaboration between the study team and the study advisory committee. Because 
implementation of different elements of competency-based learning varies across high 
schools and school districts, the study team developed separate survey modules to address 
each competency-based learning element, making it easier for users to adapt the survey to 
meet local needs. 

Developing the constructs to be measured 

Six Northeast College and Career Readiness Research Alliance members served on the 
study’s advisory committee. 4 These individuals served as content experts who advised the 
study team throughout the survey development. Ongoing collaboration between the advi- 
sory committee and the study team was critical to ensure that the survey items would elicit 
useful, valid, and reliable information about students’ beliefs about, exposure to, and under- 
standing of the key elements of competency-based learning in school districts in the region. 
The study team held several meetings with the advisory committee to generate the list of 
constructs that the survey would measure. From this list the study team developed a table of 
specifications that served as the basis for developing survey items in the next stage (see table 
1 in the main text). 

Given substantial variability in implementation of competency-based learning policies and 
practices at the school and district levels, each survey construct was addressed through a 
separate survey module. This approach allows survey users to determine which modules 
apply to their local context. Research has demonstrated that despite a growing consen- 
sus about key elements of competency-based learning, not all competency-based learning 
policies and practices require all the elements; schools and school districts often choose to 
implement only certain elements (Scheopner Torres et al., 2015; Steele et al., 2014). 

Developing and refining a pool of survey items 

The study team developed an initial pool of survey items aimed at providing complete cov- 
erage of the constructs outlined in the table of specifications. The initial item pool includ- 
ed more items than were necessary to ensure complete coverage. The advisory committee 
members then reviewed all items and gave the study team feedback. This process was used 
to evaluate the items’ face validity and content validity. 

The study team refined the item pool based on the advisory committee’s feedback. For 
example, items that were considered irrelevant or nonessential for representing the con- 
structs were removed, and the wording and format of remaining items were revised as 
needed. Decisions were made about the response options for multiple-choice and Likert- 
type items. The iterative process continued until the study team and advisory committee 
agreed that the item pool covered the constructs completely and constituted a survey of 
reasonable length (in other words, a survey that could be administered without overbur- 
dening the survey respondents or the school staff that would administer it). The refined 
item pool formed the survey’s first draft. 
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Refining the survey draft using cognitive interviewing methodology 


The study team administered the initial survey draft to seven students: four who 
were attending a Maine high school and three who were attending a New Hampshire 
high school. Different terminology is used in the two states (for example, Maine uses 
proficiency-based learning, while New Hampshire uses competency-based learning) and 
the two high schools were at different stages in the implementation of competency-based 
learning. Administering the first draft of the survey to students in two different contexts 
allowed the study team to consider how specific items performed when adapted for the 
local context. 

Cognitive interviews were conducted with the seven students. Cognitive interviewing is a 
method for identifying and correcting problems with surveys that involves administering 
a draft survey to a respondent while concurrently interviewing him or her to determine if 
the survey items are eliciting the information that the researcher intends (Beatty & Willis, 
2007). During cognitive interviews respondents were asked to think aloud as they complex 
ed the survey, thereby making explicit some of their interpretations of instructions, items, 
and response choices, as well as their decision points. 

The survey is intended to be administered online to make it easier for schools to adminis- 
ter it and calculate descriptive results, and to reduce the likelihood of errors during admin- 
istration and analysis. Therefore, cognitive interview respondents were given an online 
version of the survey that was developed using the Qualtrics online survey platform. After 
each student completed the survey, the study team used a structured protocol to gather 
detailed feedback about the student’s perceptions of the survey items (for example, under- 
standing of questions and relevance of response options) and about the survey as a whole. 

Finalizing the survey 

Information gathered during the cognitive interviews was analyzed and discussed by the 
advisory committee and study team to improve the clarity, relevance, length, and coverage 
of survey items. The study team and advisory committee worked together to determine the 
most appropriate ways to revise the survey to address the issues and concerns raised by the 
respondents during the cognitive interviewing process. At the end of this stage the study 
team produced the final survey draft and a map between the items and their constructs 
(see table 1 in the main text). 
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Appendix B. Example figures and tables 


An example of how data for item B9 might be presented as a figure is provided in figure 
Bl, while an example of how data for this item might be presented as a table is provided in 
table Bl. Brackets indicate values to be supplied by users based on survey responses. 


Figure Bl. Percentage of students selecting each statement as most true when 
asked why their school uses a competency-based grading system, 2014/15 


Teachers do a better job of grading student work 
under a competency-based grading system (n = [number]) 


Students take fewer tests under a 
competency-based grading system (n = [number]) 


for al 


A competency-based grading system makes it easier 
students to graduate from high school (n = [number]) 


A competency-based grading system provides better 
information about what a student has learned (n = [number]) 



20 40 

Percent 


Note: [Number] of [number] survey respondents indicated that a competency-based grading system was used 
at their school. These respondents were then asked, “Why do you think all or some of the courses at your 
school use a competency-based grading system?" Respondents could select only one of four options. Of the 
[number] survey respondents presented with the item represented in the figure, [number] did not respond to 
the item. 

Source: Authors' analysis of the Competency-based Learning Survey for Students. 


Table Bl. Number and percentage of students selecting each statement as most 
true when asked why their school uses a competency-based grading system, 
2014/15 


Statement 

Students responding 

Number 

“most true at my school" 

Percent 

Teachers do a better job of grading student work under a 
competency-based grading system 

[number] 

[percent] 

Students take fewer tests under a competency-based 
grading system 

[number] 

[percent] 

A competency-based grading system makes it easier for 
all students to graduate from high school 

[number] 

[percent] 

A competency-based grading system provides better 

information about what a student has learned 

[number] 

[percent] 


Note: [Number] of [number] survey respondents indicated that a competency-based grading system was 
used at their school. These respondents were then asked, “Why do you think all or some of the courses at 
your school use a competency-based grading system?” Respondents could select only one of four response 
options. Of the [number] survey respondents presented with the item represented in the figure, [number] did 
not respond to the item. 

Source: Authors' analysis of the Competency-based Learning Survey for Students. 
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An example of how data for item C2 might be presented as a figure is provided in figure 
B2, and an example of how data for this item might be presented as a table is provided in 
table B2. 


Figure B2. Percentage of students reporting exposure to specific practices and 
experiences associated with progression through demonstration of mastery, 
2014/15 


■ Never ■ Seldom Sometimes Often ■ Always Not sure 


I understand how the competencies in my 
courses will help me in the future (n = [number]) 


My teachers share examples of excellent 
work on each competency (n = [number]) 


My teachers let me know how my work will be assessed 
or graded for each competency (n = [number]) 


My teachers give me a rubric so that I know how 
I am progressing on each competency (n = [number]) 



Percent 

Note: Respondents were asked to indicate “about how often each of the following happens in courses at this 
school." 

Source: Authors' analysis of the Competency-based Learning Survey for Students. 


Table B2. Number and percentage of students reporting exposure to specific practices and 
experiences associated with progression through demonstration of mastery, 2014/15 


1 Statement 

Never 

Seldom 

Sometimes 

Often 

Always 

Not sure | 

1 understand how the competencies in my courses 
will help me in the future (n = [number]) 

[percent] 

[percent] 

[percent] 

[percent] 

[percent] 

[percent] 

My teachers share examples of excellent work on 
each competency ( n = [number]) 

[percent] 

[percent] 

[percent] 

[percent] 

[percent] 

[percent] 

My teachers let me know how my work will 
be assessed or graded for each competency 
(n = [number]) 

[percent] 

[percent] 

[percent] 

[percent] 

[percent] 

[percent] 

My teachers give me a rubric so that 1 know 
how 1 am progressing on each competency 

[percent] 

[percent] 

[percent] 

[percent] 

[percent] 

[percent] 


(n = [number]) 

Note: Respondents were asked to indicate “about how often each of the following happens in courses at this school." 
Source: Authors' analysis of the Competency-based Learning Survey for Students. 
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Notes 


1. Various terms are used to describe an approach to teaching and learning in which 
students advance after demonstrating mastery, including competency-based education, 
proficiency'based learning, mastery-based learning, and standards-based learning. This 
report uses the term competency-based learning. 

2. The two teacherdeaders are on special assignment in their district and are providing 
leadership and guidance as the district transitions to a competency-based approach to 
learning and instruction. 

3. For a copy of the complete Maine version of the survey, contact REL Northeast & 
Islands at relneiinfo@edc.org. 

4. The following individuals served on the advisory committee for this study and were 
integral to the development of the survey: Kenneth Daly (Connecticut), Lori Gunn 
(Maine), Derek Hamilton (New Hampshire), Andre Messier (Vermont), Melanie 
Montagnon (Connecticut), and Erika Stump (Maine). 
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The Regional Educational Laboratory Program produces 7 types of reports 



Making Connections 

Studies of correlational relationships 


Making an Impact 

Studies of cause and effect 


What’s Happening 

Descriptions of policies, programs, implementation status, or data trends 

o 

What’s Known 

Summaries of previous research 


Stated Briefly 

Summaries of research findings for specific audiences 

i^k 

Applied Research Methods 

Research methods for educational settings 


Tools 

Help for planning, gathering, analyzing, or reporting data or research 



